ATOM Documentation

← Back to App

E2E Test Execution Status & Issues

Summary

Running E2E tests on Fly.io deployment to identify and fix bugs.

**Current Status:** Infrastructure issues identified, working on fixes

---

Issues Found

1. ✅ FIXED: Wrong Backend URL

**Problem:** Tests were using https://atom-saas.fly.dev instead of https://atom-saas-api.fly.dev

**Fix:** Updated tests/e2e/utils/test-helpers-api.ts to use atom-saas-api.fly.dev

**Status:** FIXED - Committed in 46ac7caa

---

2. ✅ FIXED: Rate Limiting Blocking Test Requests

**Problem:** Rate limit middleware was blocking /api/test/* endpoints despite path check

**Root Cause:** The path check request.url.path.startswith("/api/test/") wasn't working reliably

**Fix:** Updated backend-saas/middleware/security.py to also check for X-Test-Secret header:

# Skip rate limiting for test endpoints (by path or secret header)
test_secret = request.headers.get("X-Test-Secret")
if request.url.path.startswith("/api/test/") or test_secret:
    return await call_next(request)

**Status:** FIXED - Committed in ddc076a2

---

3. ❌ IN PROGRESS: atom-saas-api Deployment Configuration

**Problem:** atom-saas-api machine crashes with Error: Unknown ROLE ''. Must be 'web'.

**Root Cause:** The backend-saas/fly.api.toml was missing the ROLE=web environment variable that docker-entrypoint.sh expects.

**Attempted Fix:** Added ROLE = "web" to [env] section in backend-saas/fly.api.toml

**Current Status:**

  • Updated fly.api.toml with ROLE env var
  • Dockerfile path issue: fly.api.toml points to Dockerfile.api which doesn't exist
  • Updated to use ../Dockerfile (main app Dockerfile)
  • Deployment in progress

**Remaining Work:**

  1. ✅ Add ROLE env var to fly.api.toml
  2. ✅ Update Dockerfile path to ../Dockerfile
  3. ⏳ Deploy changes to atom-saas-api
  4. ⏳ Verify machine starts successfully
  5. ⏳ Test health endpoint
  6. ⏳ Run E2E tests with rate limit bypass

---

4. ❌ BLOCKING: Dockerfile.api Missing

**Problem:** backend-saas/fly.api.toml references Dockerfile.api which doesn't exist

**Options:**

  1. Create a Python-only Dockerfile.api for atom-saas-api
  2. Continue using main Dockerfile (includes both Next.js + Python)

**Current Status:** Using main Dockerfile (../Dockerfile) as temporary fix

**Note:** The main Dockerfile works for atom-saas-api because it includes Python backend alongside Next.js

---

Test Results

Before Fixes

  • **Tests Run:** 281
  • **Passed:** 2
  • **Failed:** 279
  • **Main Error:** Rate limit exceeded

After URL Fix

  • **Tests Run:** 281
  • **Passed:** 8
  • **Failed:** 273
  • **Main Error:** Still rate limit exceeded

Expected After All Fixes

  • Should see significant improvement once rate limiting bypass is deployed
  • Many tests will likely have other issues that need fixing

---

Architecture Notes

Fly.io Apps

  1. **atom-saas** - Main production (Next.js + Python in same container)
  2. **atom-saas-api** - Python backend only (currently having startup issues)

E2E Test Infrastructure

  • Test endpoints: /api/test/* in backend-saas/api/routes/test_auth_routes.py
  • Authentication: X-Test-Secret: test-secret-key header
  • Base URL: https://atom-saas-api.fly.dev

Docker Configuration

  • **Main App:** Uses root Dockerfile with multi-stage build
  • **API App:** Should use Dockerfile.api (doesn't exist) or main Dockerfile

---

Next Steps

  1. **HIGH PRIORITY:** Deploy fixed fly.api.toml with ROLE env var
  2. **HIGH PRIORITY:** Verify atom-saas-api starts successfully
  3. **MEDIUM:** Create dedicated Dockerfile.api for Python-only backend
  4. **MEDIUM:** Run full E2E test suite and analyze failures
  5. **LOW:** Set min_machines_running=1 for atom-saas-api during testing

---

Commands Used

# Check app status
fly status -a atom-saas
fly status -a atom-saas-api

# View logs
fly logs -a atom-saas-api

# Start machine
fly machine start -a atom-saas-api <machine-id>

# Deploy
fly deploy --config backend-saas/fly.api.toml -a atom-saas-api

# Run tests
npx playwright test tests/e2e/scenarios/ --project=e2e --reporter=line

---

**Last Updated:** 2026-02-09 15:20 UTC

**Status:** Working on deployment configuration fix